Skip to content

Conversation

@msweier
Copy link
Contributor

@msweier msweier commented Oct 30, 2025

No description provided.

Copy link
Contributor

@adamscarberry adamscarberry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My only concern with a DAG that scrapes files is the payload is unpredictable. This affects trying to replay prior hours because going back to a previous hour may not produce the same results depending on when you run it. This may be unavoidable in this case, so I'm approving.

@msweier
Copy link
Contributor Author

msweier commented Oct 30, 2025

My only concern with a DAG that scrapes files is the payload is unpredictable. This affects trying to replay prior hours because going back to a previous hour may not produce the same results depending on when you run it. This may be unavoidable in this case, so I'm approving.

Yeah I think I understand that. This version should be setup to read the issue datetime in the filename and only try and get the latest issued version within the past 36 hours. So for replay it's not really going to work to get past forecasts but otherwise we were getting a lot more files.

@msweier msweier merged commit 9634497 into cwbi-test Oct 31, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants